How should we make decisions when we do not know the relevant physical probabilities? In these ambiguous situations, we cannot use our knowledge to determine expected utilities or payoffs. The traditional Bayesian answer is that we should create a probability distribution using some mix of subjective intuition and objective constraints. Imprecise Bayesians argue that this approach is inadequate for modelling ambiguity. Instead, they represent doxastic states using credal sets. Generally, insofar as we are more uncertain about the physical probability of (...) an event, there is more divergence in the credal set. Hence, their approach can represent these ambiguities via the extent of the divergence. Imprecise Bayesianism has mostly been advocated for its epistemological features. In this article, we examine its properties for decision-making. We develop a model for comparing standard and Imprecise Bayesianism by testing their performances in a classic decision problem. We find that the representational tools of Imprecise Bayesianism also cause it to underperform in our tests. This issue has been overlooked, because previous research on Imprecise Bayesianism has not utilised agent-based modelling to provide information about its performance in the short-run. Overall, we reveal the Ambiguity Dilemma for Imprecise Bayesianism: To what extent should one value representational power or decision-making performance? (shrink)

The debates between Bayesian, frequentist, and other methodologies of statistics have tended to focus on conceptual justifications, sociological arguments, or mathematical proofs of their long run properties. Both Bayesian statistics and frequentist (“classical”) statistics have strong cases on these grounds. In this article, we instead approach the debates in the “Statistics Wars” from a largely unexplored angle: simulations of different methodologies’ performance in the short to medium run. We conducted a large number of simulations using a straightforward decision problem based (...) around tossing a coin with unknown bias and then placing bets. In this simulation, we programmed four players, inspired by Bayesian statistics, frequentist statistics, Jon Williamson’s version of Objective Bayesianism, and a player who simply extrapolates from observed frequencies to general frequencies. The last player functions as a benchmark: a statistical methodology should at least outperform a crude form of induction. We focused on the performance of these methodologies in guiding the players towards good decisions. Unlike an earlier simulation study of this type, we found no systematic difference in performance between the Bayesian and frequentist players, provided the Bayesian used a flat prior and the frequentist used a low confidence level. Unlike that study, we were able to use Big Data methods to mitigate problems of random error in the simulation results. The Williamsonian player, who is a novel element of our study, also had no systematic differences in their performance, provided that they use a low confidence level. These players performed similarly even in the very short run, when players were making different decisions. Our study indicates that all three methodologies should be taken seriously by philosophers and practitioners of statistics. However, the frequentist and Williamsonian performed poorly when their confidence levels were high, and the Bayesian was surprisingly harmed by biased priors, providing some unexpected lessons for these methodologies when facing this type of decision problem. (shrink)

Calibration inductive logics are based on accepting estimates of relative frequencies, which are used to generate imprecise probabilities. In turn, these imprecise probabilities are intended to guide beliefs and decisions — a process called “calibration”. Two prominent examples are Henry E. Kyburg's system of Evidential Probability and Jon Williamson's version of Objective Bayesianism. There are many unexplored questions about these logics. How well do they perform in the short-run? Under what circumstances do they do better or worse? What is their (...) performance relative to traditional Bayesianism? -/- In this article, we develop an agent-based model of a classic binomial decision problem, including players based on variations of Evidential Probability and Objective Bayesianism. We compare the performances of these players, including against a benchmark player who uses standard Bayesian inductive logic. We find that the calibrated players can match the performance of the Bayesian player, but only with particular acceptance thresholds and decision rules. Among other points, our discussion raises some challenges for characterising “cautious” reasoning using imprecise probabilities. Thus, we demonstrate a new way of systematically comparing imprecise probability systems, and we conclude that calibration inductive logics are surprisingly promising for making decisions. (shrink)

Contemporary debates about scientific institutions and practice feature many proposed reforms. Most of these require increased efforts from scientists. But how do scientists’ incentives for effort interact? How can scientific institutions encourage scientists to invest effort in research? We explore these questions using a game-theoretic model of publication markets. We employ a base game between authors and reviewers, before assessing some of its tendencies by means of analysis and simulations. We compare how the effort expenditures of these groups interact in (...) our model under a variety of settings, such as double-blind and open review systems. We make a number of findings, including that open review can increase the effort of authors in a range of circumstances and that these effects can manifest in a policy-relevant period of time. However, we find that open review’s impact on authors’ efforts is sensitive to the strength of several other influences. (shrink)

This paper discusses the issue of overriding the right of individual consent to participation in cluster randomised trials (CRTs). We focus on CRTs testing the efficacy of non-pharmaceutical interventions. As an example, we consider school closures during the COVID-19 pandemic. In Norway, a CRT was promoted as necessary for providing the best evidence to inform pandemic management policy. However, the proposal was rejected by the Norwegian Research Ethics Committee since it would violate the requirement for individual informed consent. This sparked (...) debate about whether ethics stand in the way of evidence-based health policy, since the Norwegian Research Ethics law’s strict requirements for individual consent make it practically impossible to carry out CRTs of public health interventions. We argue that, in the case of the school closure trial, the suggested CRT would not have eliminated an epistemic gap and thus would not have justified the violation of consent rights. First, we focus on the methodological challenges to estimating quantifiable effects of school closures in the specific case of an airborne infectious disease. Second, in line with Evidential Pluralism, we highlight the value of alternative lines of evidence for informing school closure policy in a pandemic. In general, we propose that a trial requiring the waiver of participants’ consent rights must be highly likely to eliminate an epistemic gap. We elaborate on the practical aspects of this criterion and discuss the potential advantages of adding it to the Ottawa Statement on the Ethical Design and Conduct of Cluster Randomized Trials. (shrink)

Personalized medicine relies on two points: 1) causal knowledge about the possible effects of X in a given statistical population; 2) assignment of the given individual to a suitable reference class. Regarding point 1, standard approaches to causal inference are generally considered to be characterized by a trade-off between how confidently one can establish causality in any given study (internal validity) and extrapolating such knowledge to specific target groups (external validity). Regarding point 2, it is uncertain which reference class leads (...) to the most reliable inferences. -/- Instead, pharmacovigilance focuses on both elements of the individual prediction at the same time, that is, the establishment of the possible causal link between a given drug and an observed adverse event, and the identification of possible subgroups, where such links may arise. We develop an epistemic framework that exploits the joint contribution of different dimensions of evidence and allows one to deal with the reference class problem not only by relying on statistical data about covariances, but also by drawing on causal knowledge. That is, the probability that a given individual will face a given side effect, will probabilistically depend on his characteristics and the plausible causal models in which such features become relevant. The evaluation of the causal models is grounded on the available evidence and theory. (shrink)